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ABSIBACT 

Pupil activity was observed in three Title I projects 
to determine the frequency of certain essential activity categories/ 
to compare those frequencies with the supervisors* ideal/ to assess 
relationships between pupil activity and achievement/ and between 
implementation and achievement/ and to compare supervisor ratings of 
implementation with observed implementation, Besults indicated that 
pupil activity frequencies were different among the 13 to 37 sites 
within projects/ that achievement varied significantly between siteS/ 
that there was no consistent relationship between pupil activity 
variables and achievement/ and that neither measure of project 
implementation was related to achievement. Becommendations for future 
research axe made. (Author) 
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Product evaluations for three Title I elementary sehool projects in 

reading and mathematics had indicated for several years that the projects had 

been successful in raising pu^oil achievement; however, there had been no 

careful description of the procedure which had been used during the 3-5 years 

of the projects' operation. The present study was conducted in order to 

(l) develop a description of pupil activities in each project (2) test whether 

the implementation of pupil activities matched the *'ideal" of project supervisors 

(3) determine whether the frequencies of certain pupil activities were related 

to achievement at ench site {U) determine whether the degree of implementiition 

I 

to* the supervisors' ideal were related to achievement , and f5) test whether 
supervisors' ratingb of site implementation were related to achievement or 
observed implementation. 
Per s p e c t i ve s 

The rationale of other program implementation studies (Stallings and 
Kaskowitz, Siegel) was invoked here. The successful product evaluations led 
to questions regarding explanation of the observed achievement gains. ^Ihat 
was happening to these low achieving inner city children in the project class- 
rooms that might help to explain their achievement gains? Was achievement 
consistent among the 15 to hi sites within each project? Was implementation 
of the project consistent? 

The call for the step beyond description in teacher behavior studies 
(Rosenshine) was heeded when questions regarding the effect of particu3.ar 
aspects of implementation were raised. Were some aspecti3 of implementation 
more strongly related to achievement than others? What was the relationship 
between implementation to the idea], of the supervisors and achievement? 

Pupil activity was chosen as the specific component of project implementa- 
tion to be studied. The rationale for that choice was based upon several 



considerations. First, pupil activity vas one of few aspects of the project 
allowed to vnvy . Staff, materials, instructional time and content vere 
ciuite consistent within projects. Second, the model of learning which was 
\ised was that in which the response of the learner to the instructional 
setting is essential. (Anderson and Faust) Third, research literature had 
shown that some pupil activity variables were related to achievement, 
(McDonald, Stallings) Finally, low inference, objective observation of 
pupil activity was not objectionable to project staffs, but other types of 
observation were threatening to them and cooperation would have been difficu3.t 
to attain. 
Methods 

Descriptions of est^ential components of project implementation in terms 
of pupil activity were developed from project descriptions in the proposals 
for funding, observation of classroom activity and discussion with project 
teachers and supervisors. Then supervisors assigned to essential component 
categories a frequency with which the activity would be observed in a perfectly 
implemented classroom, 

A direct observation system called the Pupil Activity Profile was developed 
with four nonverbal dimensions - purpose of activity, group size, specific 
activity and interacting agent. Each dimension had between three and eight 
categories. Individual pupils were observed using a time sampling technique. 
Four observers were trained, their interobserver reliability was checked and 
deemed satisfactory at ,76 for taped observation and .87 for live observation. 
Each site was observed twice, each time by a different observer. The mean 
of the two observations for each category becrijiie the independent variables, 

AchieveirKnt measures were the adjusted site mean scores on standardized 
test subtests related to the content of the project (reading or mathematics ) , 

Supervisors rated each site on a five point scale for the degree to which 
each site implemented the project as they believed it should have been imple- 
mented. Observed implementation scores were developed by summing scores for 



each category weighted for their proximity to the ideal frequency. 

Project mean pupil activity frequencies were compared to supervisors* 
ideal mean frequencies by means of t-tests. In order to determine whether 
significant variance existed between sites in pupil achievement to make 
interpretation of correlations reasonable, and to obtain mean scores adjusted 
for pretest achievement, analyses of covarlance were conducted with posttest 
as criterion and pretest as adjuster. The observation data was analyzed for 
normality of distribution. VThere the data were usable, Pearson product- 
moment correlation coefficients were computed between site ^observation 
frequencies in certain categories of pupil activity which other research and 
literature had indicated might be related to achievement and the site adjusted 
mean achievement at each grade level. The correlations between total imple- 
mentation scores and achievement, and between supervisor ratings of implement a 
tion and achievement were also computed. 
Results and Conclusions 

All general hypotheses were rejected. Less than half of the categories 
chosen by supervisors as essential components of project implementation were 
observed to be implemented with the "ideal" frequency. 

Table 1 shows the percent of observations which included a code for each 
activity. The percentages do-ript sum to 100 even with a dimension, since 
the categories are not mutually exclusive and multiple coding was possible 
for each pupil. The dimensions of purpose and group size were most frequently 
observed to be implemented, according to the supervisors » ideal. The HILL 
Reading Project was most closely implemented to the supervisors* ideal j. 

The analysis of covariance results demonstrated that there was signifi 
cant between-site variance in all three projects at all grade levels except 

kindergarten mathematics. 

The number of significant correlations between pupil activity frequencies 
...and., achievement (five of sity^one) was at the chance-level, and. there was no 



discernible trend, even in the direction of the correlations, within projects 
or within grade levels. 

Table 2 shows the correlations found between achievement and activity. 
Four of the five significant correlations appeared in the mathematics project. 
Two of them showed a positive relationship between handling and math achievement 
at first grade. The other two were negative correlations at fourth grade. The 
proportion of instructional activity (compared to management of other activity) 
was negatively related to achievement, as was speaKing and listening acitivity. 

The same, almost random appearing results were found in the correlations 
between implementation scores and achievement and between supervisor ratings 
of implementation and achievement. 

Table 3 shows one significant correlation between achievement and 
implementation to the supervisors^ ideal - a negative correlation at Grade 
Three in one reading project. There were two significant correlations 
between supervisor rating, and achievement - both positive - at Grade 3 in 
HILL Reading and Grade 1 in mathematics. Additionally, there was a sizable 
positive correlation between achievement and supervisor rating at second 
grade in the HILL reading and positive correlations reaching the .85 significance 
level in mathematics at Grades Two and Three. 

It was concluded that (l) there was no consistent pattern of implementation 
among sites within projects, at least within the limited observation period of 
this study (2) achievement varied significantly between sites. (3) frequency of 
pupil activity variables that had been found to be positively related to pupil 
achievement in other studies did not have a consistent relationship with 
achievement- in this study with its own observation method and population {k) 
implementation of pupil activity variables to the supervisors » ideal frequencies 
not consistently related to achievement and (5) supervisors » high inference 
ratings of achievement were somewhat better predictors of achievement than the 
observation variables. ^ 
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Implication s 

Related to the findings above, the implications of the^^ study are: (l) 
Although labeled a single project, if there are considerable differences in 
implementation of pupil activities aii\ong sites, then achievement gains cannot 
be attributed to the project, but to individual sites. The project cannot 
be exported, as no single pattern of implementation can be described for use 
by others. (2) Further study of procedures used by project sites vith high 
or lo\r achievement is recommended. (3) The Title I population in this study 
may not respond to the same instructional variables as other pupil populations. 
Further study of those variables positively related to achievement in other 
studies is recommended. (h) Supervisors and program developers should 
reconsider their preferences for particular patterns of instructional activity 
by exanrlning their rationale in the light of research findings. (5) The 
actual .criteria which supervisors use in assigning high inference ratings of 
implementation to sites should be examined for possible variables for further 
research. 

The procedures followed in this study are applicable to many instructional 
programs. Tlie information regarding the activity and achievement of each site 
is useful for effective supervision. However, in the pursuit of instructional 
variables related to learning, the movement in teacher behavior studies away 
from the relatively simple correlational model to the more complex, multi- 
variate trait-treatment interaction studies should direct further research in 
this area. ' . 
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Actual and Ideal Percents of Total Observations in Which Pupils Vere 
■Engaged in Essential Categories of Pupil Activity in Three Title I Projects 



Supervisor Designated 
Essential Component 
Pupil Ac tivity CateRorie_s 

Purpose of Activity 

Instructional 

Management 
Group Size 

Independent 

One-to-one 

Small group 

Ml-group" ^ 

Activity 

Reading 

Writing ' 

Speaking 

Listening • ■ ■ 

Watching . >' ' • 

handling 
Interacting Agent 

Teacher 

Aide 

Pupil 

Instructions! Media 
Essential Corahinations 
of Categories 

One-to-one/Teacher or Aide 

Small group/Teacher or Aide 

Small group/Listening 

Independent/Reading 

Reading/Writing 

Reading/Listening 
■ SpeaXirig/Listening 

Listening/Watching 

Listening/Pupil 



Reading Center 
Ideal Actual 



26. T 

•T 

15.0 
50.0 



20.0 
26.T 
15.0 

1+0.0 
10.0 

13.3 

10.0 

13.3 
10.0 



52.9 
15.1 
30.0 
80.1 



12.1 
T.5 
IT.5 
11.9* 
kl.O 

33.2 
18.8 
23.8 



PROJECT 

HILL Reading 
Ideal Actual 



88.0 


02.1 


T6.0 


T5.6* 


12.3 


IT. 8 


22.3 


23.0* 




25.8* 


53.3 




■20.0 


20.6* 


20.0 


21.9*' 


2I1.3 


18.5* 


IT.T 


20.9* 


2T.T 


^1.1 


,9.0- 


-"1-1-.-5*— - 


I16.T 


6T.5 


I16.T 


63.1 


10.0 


13.T 


13.3' 


19.8 


26.T 


1*5.8 


11. T 


33.8 


33.3 


llj.l 


26.T 


liT.6 


6.T 


25.9 


6.T 


11.5 



21.T 
13.3 
10.0 
60.0 



22.1* 
16.5* 
IT.3 
80.5 



■ v> difference vas not significant (t value with p>. 05) . _ , 

ERiCNote,: Percentages do not sum to lOOl hecause of multiple coding of each pupil. 



Mathematics 
Ideal Actual 



95.0 9^.3*. 



20.0 13. 
T5.0 91.'T 



20.0 
20.0 
20.0 
80.0 
50.0 
20.0 



69.T 
26.lt 
50. T. 
85.T 
I19.O* 

I1O.9 



25.0 25.lt* 

50.0 60.2 

25.0 I13.3 

50.0 90.2 



30.0 


l^.T 


25.0 


12.9 


20.0 


8.8 






10.0 


12.3* 






21.0 


35.8 




26.1 


20.0 


19.0* 


10.0 


20.0 


26. f 


5.0 


5T.3 


10.0 


1^.8 ■ 


25.0 


^5.9 


20.0 


8.T 


50.0 




10.0 ■ 


T.T* 


25.0 


39.T 
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TABLE 2 



Correlations Between Certain Pupil Activity Fr5q.uencies 
and Adjusted Site Mean Achievement in Three' Title I Projects 



!ategory of Pupil Activity , 
Hypothesized to he 
Related to Achievement 



Reading Center 
GRADE Two Three Four 
]j(Sites) 26 25 18 



total Instructional 
[ Independent Reading' 
Potal I Reading 
[ One-to-one/Teacher or 
Aide 

E Reading/Writing 
I Reading/SpeaXing 
I Reafding/Listening 
I Reading/Handling 
r Listening/SpeaKing 
I Listening/Watching ^ 
I Listening/Handling . 



-.28 


.1+2* 


-.35 


.25 


-.16 


.03 


NT 


NT ■■ 


.... 


.18 


.06 


.09 


.111 


-.02 


.11; 


.05 


.16 


.16 


-.11 


-.18 


■1? 


,11] 


-.11 


.08 


iit' 


NT 


NT 


■IT 


-.28 


'.16 


-.05 


.22 


.11 


.32 


-.12 




.05 


.03 


-.23 


-.l;9 


.20 


.IT 



PROJECT 
HILL Reading 
Two Three Four 
6 13 10 



Mathematics 
One Two Three" Four 



25 


33 


33 


33 


.12 


■IV 


-.OT 


-.36* 


.06 


-.25 


.12 


.08 


.03 


-.09 


.06 


.10 


.35* 


.05 


.10 


.01 


.11 


-.18 


.06 


-.50- 


■.22 


.16 


-.13 


.08 


.38* 


.03 


.09 


.13 



*p<^.10 

IIT-..= ..not tested due to non-normal distribution of dependent variable 



TABLK 3 

Correlation Coefficients Betveen Adjusted Site Mean Achievement, Observed 
lEplenentation and Supervisor Rating of Implementation in Three Title I Projects 



Project 



Reading Center 



N= 



HILL Reading 



Mathematics 



Grade Level One 



-.16 

25 



.23 

26 



.09 
6 



.01 
33 



Three Four 



..38* 
25 



',11 
13 



,00 
33 



.3B 

18 



-.25 

10 



.IT 
33 



Achievement and 
Supervisor Rating of 
Site Impltnientation 



Observed 

Implementation;, 
and Supervisor; 
Rating 




